"Outbound" Azure Integration: CRM can post Execution Context to CRM-aware Azure Listener

• Features / functionality:
  • Dynamics CRM can send event and entity details to an Azure endpoint. The data, by default, is contained in a CRM RemoteExecutionContext object. A CRM Service Endpoint is configured to send the context to the Azure endpoint.
  • Integrate with Microsoft Dynamics CRM (CRM) and Microsoft Azure by coupling the CRM event execution pipeline to the Windows Azure Service Bus (ASB). In essence, the CRM pipeline connects to the ASB enabling the data that has been processed as part of the current CRM operation (create, update, etc.) to be posted to the bus.
  • Developers can register plug-ins with CRM that can pass run-time message data, known as the execution context, to one or more Azure solutions in the cloud.
  • ASB solutions (plug-ins) that are “CRM aware” can listen for and read the data that is posted on the service bus by CRM. The posted data is stored in a RemoteExecutionContext class instance that is an extended version of IExecutionContext passed at run time to CRM asynchronous plug-ins.
  • CRM can send messages to the ASB in these ways: One-way, Two-way, REST, Queue, and Topic.
  • The Windows Azure Service Bus combined with Shared Access Signatures (SAS) or the Windows Azure Access Control Service (ACS) provides a secure communication channel for CRM run-time data to external line of business applications.
• Key elements:
  • CRM Asynchronous service (async service) is responsible for posting the CRM remote execution context to the ASB. Each post is performed by a system job of the asynchronous service. A user can view the status of each system job using the Microsoft Dynamics CRM web application.
  • CRM Service Endpoint: Created with the CRM plug-in registration tool. Defines the Azure endpoint and SAS or ACS configuration details. (Note: A Service Endpoint that is configured for an Azure Service Bus essentially takes over the Service Bus. Thus, it's currently not possible for Service Endpoints from different CRM organizations to send messages to the same Azure Service Bus.)
  • Plug-ins and custom workflow activities: There are two kinds of asynchronous registered plug-ins supported by the integration feature: out-of-box (OOB), and custom.
    • Out-of-box Azure aware plug-in: Microsoft provides a pre-coded plug-in that, when registered for an entity and message (e.g., Create of Account), it can post the execution context to the Azure Service Bus. It executes in full-trust; Can notify the async service to post the current request's context to the ASB. A developer needs to register a step on this plug-in that identifies the target message and entity; Doesn't (can't) call any CRM SDK methods because it's a pre-coded assembly.
      • Limitations:
        • Cannot call CRM SDK methods for any purpose, such as retrieving additional data, validating data, creating records, etc.
        • Cannot write custom trace statements for purposes of auditing, logging and troubleshooting
        • Can only run asynchronously
        • Can’t be used with a workflow, at least not directly
    • Custom plug-in: Executes in partial trust in the sandbox; Can call any CRM SDK methods. Can initiate posting CRM context to the service bus; See CRM SDK for sample Azure-aware code
    • Custom Workflow Activities: See "Custom plug-in above"; The same general details apply
• Prerequisites: Get a public certificate (from CRM Online or from an issuing authority); Download and install the Azure SDK and read the documentation; A Windows Azure account with service bus access must be purchased.
• Definitions:
  • Windows Azure Service Bus: The service bus relays the remote execution context between CRM and ASB solution listeners. The Windows Azure Access Control Service (ACS) manages claims based authentication security. See "Overview of Windows Azure Service Bus" below.
  • Relay Endpoint Contract: Listener is "CRM aware", actively listening. If not, the message is lost.
  • Queue Endpoint Contract: Listener does not need to be actively listening. A queue stores the message. The listener (also known as a consumer or receiver) receives the next message for processing.
• A listener is made “CRM aware” by linking it to the Microsoft.Xrm.Sdk assembly so that type RemoteExecutionContext is defined and available.
• Certificate details for on-premises and IFD:
  • For CRM 2013 on-premises and IFD installations, you can purchase a private certificate from an issuing authority. Import the certificate file into the Personal\Certificates store on your computer using the certificate Microsoft Management Console (MMC) snap-in. Next, export a public key file of your certificate in Base64 format.
  • CRM Online comes pre-configured to work with Azure.

Azure Code/Project Samples in the CRM SDK

Listed below are the sample projects in the CRM 2013/2015 SDK for Azure integration. It is assumed that you have an Azure account and CRM and Azure are configured for integration. You also need to know the service's (listener's) namespace (see "Create a Service Namespace") to call the CreateServiceUri of the ServiceBusEnvironment object.

• OneWayListener
  • This sample listener application registers a remote service plug-in that executes whenever a CRM message is posted to a one-way (no return value) endpoint on the Windows Azure Service Bus. When the plug-in executes, it outputs to the console the contents of the CRM execution context contained in the message.
  • Deployment: See Walkthrough: Register an Azure-Aware Plug-in with Plug-in Registration Tool. See the readme for this example for settings that are unique to this example.
• TwoWayListener
  • This sample registers a remote service plug-in that executes whenever a CRM message is posted to a two-way (return value) endpoint on the Windows Azure Service Bus. When the plug-in executes, it prints to the console the contents of the CRM execution context contained in the message. For this two-way contract, the Execute method returns a string from the method call.
  • The plug-in implements interface ITwoWayServiceEndpointPlugin
  • When using the out-of-box (ServiceBusPlugin) plug-in with a two-way or REST listener, any string data returned from the listener is not used by the plug-in. However, a custom Azure aware plug-in could make use of this information.
• RestListener
• QueuedListener (Note: Microsoft indicated that the "message buffer" approach used in this example is deprecated; they recommend using the Persistent Queue Listener.)
• PersistentQueueListener
  • This sample shows how to write a Windows Azure Service Bus listener application for a persistent queue endpoint contract. The listener waits for a CRM message to be posted to the service bus and to be available in the endpoint queue. When a message is available in the queue, the listener reads the message, prints the CRM execution context contained in the message to the console, and deletes the message from the queue
• Plug-ins
• AzureAwareWorkflowActivity

One Way Listener Diagram

This diagram illustrates the functionality in the "OneWayListener.csproj" project that Microsoft provides in the Dynamics CRM 2013 SDK.

Persistent Queue Listener Diagram

This diagram illustrates the functionality in the "PersistentQueueListener.csproj" project that Microsoft provides in the Dynamics CRM 2013 SDK.

References / Links

Azure

What is Windows Azure?
Windows Azure Documentation
Introduction to Windows Azure integration with Microsoft Dynamics CRM

• Includes a diagram that shows a listener application running behind a company's firewall that gets messages posted by CRM based on an CRUD operation.

Overview of Service Bus Messaging Patterns
Windows Azure SDK Downloads
Remote debugging Windows Azure Cloud Services from Visual Studio 2013

Azure-CRM Integration

Configure Windows Azure integration with Microsoft Dynamics CRM

Walkthrough: Register an Azure-Aware Plug-in with Plug-in Registration Tool

• This tutorial covers how to configure a Service Endpoint including settings Service Namespace, Path, Contract, Claim, ACS, etc.

General Integration Information

Integration Patterns

Azure SDK (Windows, .NET, Visual Studio)

• The SDK installs to C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK by default
• Adds assemblies to the GAC, templates to Visual Studio (CloudService), MS Build extensions, AzCopy tool, remote debugger, emulator, etc.

Overview of Windows Azure Service Bus

Windows Azure Service Bus supports both basic queuing and publish-and-subscribe forms of messaging. The publish-and-subscribe approach enables one piece of code to send a message on a topic and have multiple other pieces create subscriptions to the topic. Service Bus also includes a relay service that enables application components to securely communicate through firewalls by exchanging messages through an endpoint hosted in the cloud. In Windows Azure terminology, an endpoint is one or more ports by which you can directly access an individual instance of a service. Application code communicates with Windows Azure services by binding to endpoints. Service Bus thus enables Windows Azure applications to communicate with other Windows Azure applications, with applications running on some other cloud platform, or even with applications running outside the cloud.

Real-world example of Using Microsoft Azure and Dynamics CRM Together: Scheduled Workflows

Execute Dynamics CRM Workflows on a Schedule

At Altriva, we utilized Azure services to schedule and execute workflows in Dynamics CRM Online. This solution is sometimes described as "batch workflow", "bulk workflow", or "scheduled workflow" execution. Here's how it works. Within CRM, the end-user (usually a CRM admin) can specify the workflow to run and the FetchXml query to determine the records on which the workflow will run. The user then sets up (also within CRM) the schedule for when to run the batch workflow job. This sends a message to an Azure Service Bus queue via a CRM Service Endpoint. An Azure worker role gathers the schedule details from CRM and sets up an Azure Scheduler job. Lastly, when the Azure Scheduler job's date/time/interval is reached, the worker role picks up a message from a storage queue and then runs the workflow for the target CRM records.

This also make bulk updates in CRM very easy. You can create a workflow process that performs the action(s) you need, generate the FetchXml to define the records on which the workflow needs to run and set up a one-time scheduled job. Azure Scheduler will fire an event right away triggering the code that executes the workflow on each record returned by the FetchXml query. This eliminates the need to run workflows manually 250 records at a time or writing a one-off console app to perform a bulk update.

Other uses for scheduled workflows include:

• Send e-mails based on a schedule
• Push or pull data between another system based on a schedule
• Run a set of custom data clean-up workflows or plug-ins on a periodic basis
• Use in combination with CRM 2015 Change Tracking to send changes to a different system on a regular basis
• Update a set of records (based on a FetchXml query) each day
• Schedule a workflow that calls a workflow activity that creates a PDF-based report and e-mails it out at the start of each day.
• Bulk update thousands of records on demand.
• Reduce or eliminate the need for "waiting" workflows (workflows that go into a wait state until an event occurs).
• Call a web service each day to update data from external data sources. For example, update new accounts and contacts with time zone and lat/long values.
• Enable/disable workflows or plug-in steps on a schedule. For example, you might want to disable a workflow during weekends.
• Check the due date for all open Cases once a day and set ‘Exceeding SLA’ indication if due date has passed
• Send monthly news letter automatically to target customers
• Update CRM Users details from Active Directory once a day
• Once a month, disqualify all leads with no open activities
• Set a schedule to update accounts, contact and leads with InsideView data.

Service Bus Messaging inbound to Dynamics CRM

Another way to use Microsoft Azure with Dynamics CRM is to set up an Azure Service Bus Queue or Topic and send messages to it from systems that have data that's needed in CRM. An Azure worker role can listen for messages, decode them, map the values to CRM entities and fields and then perform the necessary CRUD operations within CRM.

CRM Azure Service Endpoint (ServiceEndpoint)

ACS Configuration

After clicking the "Configure ACS" for a Service Endpoint (in the plug-in registration tool), where the ACS configuration succeeded, the following is an example of what's shown in the field below the "Configure ACS" button.

Trying to find out the ACS Version.
ACS Version is: V2

Begin Configuring For Service Bus Scope

Creating ManagementService for contoso-bus-dev-sb
Created ManagementService and cleared the cached token
Found ServiceIdentity with Name: ContosoDev, ID: 37156768
Found Issuer with Name: ContosoDev, ID: 31056766
Found RelyingParty with Name: CrmBatchWorkflowSchedContext, ID: 28104939
Deleted RuleGroup with Name: Rule group for CrmBatchWorkflowSchedContext (Only if existed)
Created RuleGroup with Name: Rule group for CrmBatchWorkflowSchedContext
Assigned RuleGroup to RelyingParty
Created Rule: sampleorgsendrule
Created Rule: sampleownersendrule
Created Rule: sampleownerlistenrule
Created Rule: sampleownermanagerule

End Configuring For Service Bus Scope

Examples of Combining Azure and Dynamics CRM (by Azure Feature/Service)

Last updated: 8/6/2015

Azure Category	Azure Feature / Service	Brief Description	Usage/Scenario Examples with Dynamics CRM
Compute	Virtual Machines	An Azure virtual machine gives you the flexibility of virtualization without spending the time and money to buy and maintain the hardware that hosts the virtual machine. FAQ	- Build a CRM environment using an Azure VM (for dev/test purposes) - Run a data quality tool on a VM and update Dynamics CRM with the result - Install SQL Server Reporting Services (SSRS) and use the service to create reports from CRM data that was copied to SQL Server. This would allow for full SSRS capabilities including scheduling reports and using SQL to query data.
Compute	Cloud Services	Deploy highly-available, infinitely-scalable applications and APIs	- Build a worker role application that retrieves CRM RemoteExecutionContext objects (as sent from CRM through a Service Endpoint) from a service bus queue and updates a separate system according to the retrieved data. - Build a service (worker role application) that retrieves JSON-based messages, maps the data to CRM entities/fields and performs CRUD operations in CRM. - Build a service that runs on a schedule (cron job) to perform regular actions on CRM data or metadata. Use Azure Scheduler to initiate the tasks.
Compute	Batch	Cloud-scale job scheduling and compute management	- Schedule and run very large batch data operations and summarize the data in Dynamics CRM for use on dashboards, charts, reports and to create business processes (e.g., tasks, work queues) that allow end-users to take action on the resulting data from the Azure Batch processes.
Compute	RemoteApp	Azure RemoteApp helps you provide secure, remote access to applications from many different user devices. FAQ	- Example: Set up Azure RemoteApp to allow your sales team, who also use CRM online, to access a custom quoting and pricing application that runs on a Windows Server (either in Azure or local network) from a wide range of devices.
Web & Mobile	App Service	Create web and mobile apps for any platform and any device
Web & Mobile	Web Apps	Deploy and scale websites and web apps	- You need to provide data residing in a SQL Server database on a CRM form. You create an Azure website that is designed to take in the CRM record's GUID as a parameter, query the SQL database and provide the results for the user. (See "Implement single sign-on from an ASPX webpage or IFRAME")
Web & Mobile	Mobile Apps	Build iOS, Android, and Windows apps	- Track mobile app users (customers) in CRM. Design the mobile app to send data to Azure (e.g., to a queue) for processing and storage in Dynamics CRM. - Push CRM record changes, such as a finalized knowledge base article, to mobile subscribers of the data.
Web & Mobile	Logic Apps	Automate the access and use of data across clouds without writing code	- Use the Dynamics CRM connector to read/write CRM data.
Web & Mobile	API Apps	Use ASP.NET, Java, PHP, Node.js or Python to build new API Apps.
Web & Mobile	API Management	Publish any API to developers and partners
Web & Mobile	Notification Hubs	Send push notifications to any platform, from any backend
Web & Mobile	Mobile Engagement	Microsoft Azure Mobile Engagement is a data-driven user engagement platform that combines real-time analytics with push notifications and in-app messaging to deliver the right message, to the right end-user, at the right time and location. Mobile Engagement closes the marketing loop for app developers and marketers, giving them a more complete ROI picture with better visibility into customer acquisition costs, retention, and ARPU.
Web & Mobile	Azure Functions	Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in Azure or third party service as well as on-premises systems.	- Run periodic (scheduled) jobs in CRM, such as running a workflow for one or more records on a regular basis. - React to events in CRM using a Service Endpoint and the provided RemoteExecutionContext. - Provide on-demand and scheduled data roll-ups (aggregation). - Package CRM metadata into a serialized and portable format. Then, create Azure Function wrappers for that cached metadata to provide to Logic Apps the ability to get a wider range of CRM metadata beyond picklists, such as entity and field names and properties, system users, FetchXml queries (saved views), queues, etc. This would allow for much faster access to CRM metadata vs retrieving it in each Function or Logic App execution.
Data & Storage	SQL Database	Microsoft Azure SQL Database (Azure SQL Database) is a relational database-as-a-service, which falls into the industry category Platform as a Service (PaaS). Azure SQL Database is built on standardized hardware and software that is owned, hosted, and maintained by Microsoft. With SQL Database, you can develop directly on the service using built-in features and functionality. When using SQL Database, you pay-as-you-go with options to scale up or out for greater power.	- Store CRM Online data in your own Azure SQL database for advanced queries, reporting, analytics, backup, integration.
Data & Storage	DocumentDB	Azure DocumentDB is a NoSQL document database service designed from the ground up to natively support JSON and JavaScript directly inside the database engine.	- Store and retrieve non-relational "on the fly" data. For example, store data that is not needed in CRM queries, views, etc. but needs to be related to CRM records. Examples include survey results, miscellaneous company details, custom audit information, search keywords, etc.
Data & Storage	Redis Cache	Microsoft Azure Redis Cache is based on the popular open source Redis Cache. It gives you access to a secure, dedicated Redis cache, managed by Microsoft. A cache created using Azure Redis Cache is accessible from any application within Microsoft Azure.	- Store CRM metadata in Redis Cache. Multiple Azure cloud services (worker roles) use this cache while processing data to validate data based on CRM field data type, picklist values, field lengths, etc.
Data & Storage	Storage	Azure Storage provides the flexibility to store and retrieve large amounts of unstructured data, such as documents and media files with Azure Blobs; structured nosql based data with Azure Tables; reliable messages with Azure Queues, and use SMB based Azure Files for migrating on-premises applications to the cloud.	- Offload file attachments from the Dynamics CRM database to Azure Storage. Provide a custom UI within CRM (in an IFRAME) to list available documents. This reduces CRM storage requirements and puts documents in a central location ready for indexing, searching, versioning, maintenance, etc.
Data & Storage	StorSimple	StorSimple is a cloud-integrated storage (CiS) product that allows you to store commonly used data on premises and put more infrequently accessed data in Azure.
Data & Storage	Azure Search	Azure Search Service is a fully managed cloud service that allows developers to build rich search applications using a .NET SDK or REST APIs.	- Send all notes (annotations) to an Azure service bus queue. An Azure worker role sends the data to Azure Search. CRM users search for content within an IFRAME on a dashboard. - Move attachments from the CRM database, store in Azure Storage. Extract the text from documents using IFilter. Search that text using Azure Search. - Aggregate (e.g., using Azure Data Factory) survey responses, feedback, social media content etc. and provide a full-text search capability within Dynamics CRM to help facilitate decision making.
Analytics	HDInsight	Azure HDInsight deploys and provisions Apache Hadoop clusters in the cloud, providing a software framework designed to manage, analyze, and report on big data.
Analytics	Machine Learning	Azure Machine Learning is a fully managed service that you can use to create, test, operate, and manage predictive analytic solutions in the cloud.	- Predict opportunity win/loss probability for a prospect based on existing CRM data.
Analytics	Stream Analytics	Azure Stream Analytics is a fully managed real-time stream computation service hosted in Microsoft Azure, which provides highly resilient, low latency, and scalable complex event processing of streaming data. Azure Stream Analytics enables developers to easily combine streams of data with historic records or reference data to derive business insights easily and quickly.	- Issue alerts when customer experience within a time frame is degraded
Analytics	Data Factory	Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines using structured, semi-structures and unstructured data sources.	- Aggregate survey responses, feedback, social media content etc. and provide a full-text search capability (e.g., using Azure Search) within Dynamics CRM to help facilitate decision making.
Analytics	Event Hubs	Event Hubs is a highly scalable ingestion system that can process millions of events per second, enabling your application to process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs, you can transform and store data using any real-time analytics provider or storage cluster.
Analytics	SQL Data Warehouse	An elastic data warehouse-as-a-service with enterprise-class features.
Analytics	Data Catalog	An enterprise-wide metadata catalog that makes data source discovery easier. It’s a fully managed service that enables any user – from analysts to data scientists to data developers – to register, enrich, discover, understand, and consume data sources.
Networking	Virtual Network
Networking	ExpressRoute
Networking	Traffic Manager
Media & CDN	Media Services
Media & CDN	CDN
Hybrid Integration	BizTalk Services
Hybrid Integration	Service Bus		- Send messages from CRM to a service bus queue or topic using a ServiceEndpoint. Azure worker role repackages the data into JSON and sends it to another queue for downstream processing. - Azure worker role receives messages from a service bus queue, maps the data to CRM entities and field and performs CRUD operations in CRM.
Hybrid Integration	Backup
Hybrid Integration	Site Recovery
Identity & Access Management	Azure Active Directory
Identity & Access Management	Multi-Factor Authentication
Developer Services	Visual Studio Online	Visual Studio Online provides hosted source control, work item tracking, agile planning, build and load testing services.	- Use TFS to maintain your CRM source code (plug-ins, web resources, documentation, etc.) - Export your CRM metadata and store it in TFS to maintain a history of changes and to perform diffs on the metadata - Manage your CRM configuration and development projects
Developer Services	Application Insights
Management	Preview Portal
Management	Scheduler		- Have an Azure worker role perform operations in CRM when the Azure Scheduler job sends a message to a queue.
Management	Automation
Management	Operational Insights
Management	Key Vault		- Use this service to store highly sensitive data outside of Dynamics CRM. Provide links to the data within CRM (i.e., on a form) and lock it down with a password.

.